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Introduction 


Since  2004  the  authors  have  been  actively  involved  in  the  innovative  Tactical  Network 
Topology  (TNT)  experimentation  program,  which  the  Naval  Postgraduate  conducts  quarterly 
with  the  United  States  Special  Operations  Command  (USSOCOM)  to  explore  emerging  agile 
adaptive  tactical  networks.  One  of  our  first  findings  was  a  set  of  solutions  enabling  rapid 
adaptation  of  broadband  wireless  networks  to  the  commander’s  needs,  which  we  have  named 
Network-on-Target  (Bordetsky  and  Bourakov,  2006).  According  to  the  Network-on-Target 
(NoT)  operational  concept  the  adaptation  starts  at  the  level  of  the  situational  awareness  interface 
used  by  the  local  commander.  When  the  commander  desires  to  have  one  of  the  mobile  nodes 
(operators,  vehicles,  unmanned  platforms)  move  to  a  target  location  which  is  beyond  the  current 
network  range,  he  drags  the  selected  node  icon  on  his  display  to  the  target  position  .  In  response, 
the  self-aligning  robotic  antennas  autonomously  adjust  their  orientation  to  establish  and  support 
the  short-term  network  links  between  mobile  nodes. 

Our  subsequent  experiments  with  agile  adaptive  networking  on  the  move  at  high-speed  and 
through  rugged  hazardous  terrain  (NPS  MIO  08-4  After  Action  Report,  2008)  showed 
significant  limitations  when  using  visual  representation,  i.e.  when  using  computerized  viewing  of 
the  common  operational  picture  as  the  main  human-computer  interface  for  developing 
situational  awareness  and  remotely  controlling  networking  robotic  nodes. 

In  operations  involving  small  vessel  interdiction  at  high  speeds  of  30-50  nautical  miles  per  hour, 
the  remote  control  of  Unmanned  Aerial  Vehicles  (UAVs)  while  moving  rapidly  through  rugged 
terrain,  or  conducting  casualty  assistance  while  still  in  the  hostile  area,  even  the  most 
experienced  operators  have  no  chance  to  concentrate  on  opening  and  viewing  a  computer  screen 
with  the  map-based  situational  awareness  interface. 

Unlike  the  common  operational  picture  view,  rich  in  content  voice  communication  doesn’t 
interfere  with  the  operators  ability  to  navigate  and  focus  visually  on  the  target.  This  represents 
|  almost  the_only  feasible  solution  for  getting  the  situational  awareness  messages,  adjusting  the 
position  and  orientation  of  robotic  units,  or  operating  unmanned  vehicles  and  remote  sensors, 
while  keeping  hands  and  eyes  free  for  split  second  actions. 

The  Voice-on-Target  (VoT)  approach  extends  the  earlier  NoT  concept  into  the  new  unexplored 
and  very  promising  field  of  the  unified  voice  communication  interface  to  robotic  adaptive 
elements  of  the  emerging  tactical  network-centric  environment.  The  paper  describes  first 
groundbreaking  results  in  developing  the  VoT  architecture,  the  tactical  portal,  and  field 
experimentation  with  designed  solutions. 


1.  Voice-on-Target:  Concept  and  Portal  Architecture 

The  last  decade  has  shown  advances  in  VoiceXML,  CCXML,  CallXML,  and  other  voice 
controllable  Internet  surfing  techniques,  with  the  latest  applications  created  by  the  telephony 
community.  Together  they  provide  a  unique  background  for  the  new  research  dimension,  voice 
control  of  a  computer’s  peripheral  infrastructure,  robots,  and  sensors. 

The  most  generic  approach  utilizing  voice  controllable  robots  for  military  applications  is 
presented  on  Figure  1.  It  highlights  the  core  elements  of  Voice  Portal  Infrastructure.  The  Voice 
Portal  is  considered  herein  as  an  addition  to  the  well-known  and  widely  used  network  enabled 
robotic  systems.  The  voice  command  may  be  delivered  to  the  robot  for  execution  either  over  a 
wide  area  network  connection,  like  the  Internet,  or  from  within  the  tactical  local  area  network 
infrastructure.  The  use  of  a  combination  of  both  types  of  networks  may  be  very  beneficial  since 
it  brings  a  global  reach  capability  to  the  tactical  level  of  robotic  systems  controlled  by  voice 
commands. 

Any  commercial  cell  phone  can  be  used  as  a  voice  terminal  to  provide  a  communication 
interface  to  the  robot.  The  regular  ground  line  telephone,  VoIP,  and  Soft  Phone  may  also  be  used 
as  a  voice  terminal  device.  Another  new  element,  employed  in  the  Voice  Portal  infrastructure, 
is  a  computer  system  to  support  voice  specific  services.  The  basic  set  of  services  provides  two- 
way  voice  communication  and  includes  Session  Initiation  Protocol  (SIP),  Voice  recognition,  and 
Text-to-Speech  (TTS)  service.  Literally,  implementation  of  such  a  system  allows  an  to  operator 
to  “talk  to  robot”.  As  a  result,  the  operator  will  keep  his  hands  free,  and  not  preoccupy  his  vision 
with  the  computer’s  graphical  user  interface  (GUI). 
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Figure  1.  Voice-on-Target  Portal  Infrastructure 


Hyper  Speech  is  defined  as  a  voice  hyperlink  to  navigate  fragments  of  voice  application,  and 
provides  voice  browsing  between  different  voice  applications.  Like  any  hyperlink,  it  offers  a 
great  flexibility  and  immense  usability  to  voice  applications.  When  combined  with  seamless 
XML-based  protocol,  such  as  VoiceXML,  voice  application  allows  development  and 
implementation  of  simple  but  practical  voice  controlled  robotic  systems.  The  Operation  Support 
and  Situation  Awareness  Server  (SA  Server),  shown  on  Figure  1,  is  employed  to  integrate  Hyper 
Speech  navigation,  VoiceXML  media  delivery  protocol,  and  logic  of  robotic  system  control  into 
a  single  voice  application  package. 


An  example  of  voice  dialogue  is  presented  on  Figure  2.  The  user  initiates  a  voice  navigation 
process  by  placing  a  phone  call  to  the  system.  Depending  upon  the  user’s  selection,  voice 
applications  running  on  the  SA  Server  collect  all  necessary  information  for  task  execution  with 
additional  voice  dialogue.  Then  it  asks  conformation  for  tasking  and  sends  the  control  message 
to  the  robot  to  proceed  with  the  assigned  task.  The  Status  Request  will  initiate  robot  and  SA 
Server  database  queries  to  collect  current  and  history  log  data  and  deliver  it  back  to  the  user  in 
voice  format.  That  is  what  might  be  considered  as  a  “talk  to  robot”  mechanism. 
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Figure  2.  Voice  Control  diagram 


The  following  on  examples  describe  the  first  successful  field  experiments  with  the  VoT  portal. 


2.  Field  Experiment:  Applying  VoT  to  a  Battlefield  Medical  Networking 
Scenario 

One  of  the  first  successful  applications  of  the  VoT  solution  took  place  during  the  Tactical 
Network  Topology  (TNT)  Battlefield  Medical  Networking  Experiment.  In  this  particular 
discovery  and  constraints  analysis  experiment  the  VoT  solution  was  used  to  explore  the 
feasibility  of  directing  an  Unmanned  Aerial  Vehicle  (UAV)  to  a  casualty  site,  take  images,  and 
activate  a  drug  delivery  micro  device  (a  prototype  of  the  future  battle  suit  nanotechnology 
patch). 


The  device,  developed  at  the  MIT  Institute  for  Soldier  Nanotechnology  was  integrated  with  the 
TNT  network  and  set  up  for  being  activated  by  one  of  the  following  methods: 

•  through  the  tactical  network  over  the  TNT  Situational  Awareness  (SA)  interface; 

•  by  commands  sent  over  the  commercial  GPRS  cell  phone  network; 

•  by  voice  command  sent  to  the  CENETIX  (Center  for  Network  Innovation  and 
Experimentation )  Voice  Portal  over  a  commercial  cellular  network. 

Figure  3  illustrates  the  network-controlled  nano  sensor  setup.  The  drug  delivery  actuator  was 
placed  on  a  simulated  casualty  (mannequin). 


Figure  3.  Network-controlled  drug  actuator  setup 

The  experiment  was  conducted  in  accordance  with  the  following  experimentation  steps: 

Step  1 :  The  casualty  (mannequin)  was  positioned  at  the  remote  area.  The  person  acting  as  the 
battlefield  medic  was  located  at  the  Light  Reconnaissance  Vehicle  (LRV)  site  which  was 
forward  deployed  close  to  the  casualty  location.  The  LRV  was  connected  to  the  tactical  network 
via  a  broadband  wireless  link.  An  e-tag  reader  was  placed  with  the  medic  in  the  LRV.  The  GPS 
position  of  the  e-tag  reader  provided  the  casualty  position.  The  e-tag  health  data  was  transferred 
via  a  Bluetooth  link  to  the  e-tag  reader,  and  then  propagated  further  via  the  GPRS  link  to  the 
medical  data  base  in  a  remote  location. 

Step  2:  The  UAS  was  tasked  to  fly  to  the  casualty  site  by  the  battlefield  medic  via  a  cellular 
phone  GPRS  interface,  and  by  the  medic’s  voice  command  over  the  VoT  Portal. 

Step  3:  A  UAS  onboard  high  resolution  camera  used  to  take  a  picture  of  the  casualty  and  deliver 
it  to  the  Tactical  Operations  Center  (TOC)  and  to  the  forward  deployed  medic’s  cell  phone. 

Step  4:  The  drug  delivery  device  was  activated  via  the  GPRS  network  from  the  medical 
commander’s  cellular  phone,  and  over  the  Voice  Portal. 


In  accordance  with  the  described  plan  the  experimentation  team  successfully  initiated  delivery  of 
liquid  drug  release  into  a  liquid  vial  worn  on  the  casualty’s  life  vest  by  sending  a  drug  release 
voice  command  over  the  CENETIX  Voice  Portal.  This  was  repeated  again  by  sending  a 
command  over  a  GPRS  wireless  cellular  phone  (Blackberry  handheld)  device.  After  each  voice 
command  activation,  the  medic  provided  voice  comments  to  be  recorded  in  Observer  Notepad 
over  Voice  Portal  for  TOC  feedback  on  his  actions. 


Figure  4.  TOC  Video  Wall  view  of  the  casualty  simulation  mannequin  and  first  release  of  liquid 
drug  into  vial  (up-right  comer  of  picture)  upon  voice  activation  from  the  medic’s  Blackberry 

handheld. 

The  NPS  UAS  which  was  controlled  by  voice  commands  successfully  captured  a  high  resolution 
image  of  the  casualty  location  during  flyover  of  the  casualty,  and  successfully  relayed  this  high 
resolution  image  to  the  TOC  video  Wall. 


Figure  5.  Casualty  e-tag  alert  activation  and  propagation  to  the  shared  SA  view. 


Figure  6.  Battlefield  medic  providing  voice  control  of  UAS  medicine  injection  over  the  VoT 

interface 


Figure  7.  Casualty  site  overlooking  voice-controlled  camera  on  top  of  medic’s  LRV 


Figure  8.  High  resolution  imagery  of  the  casualty  site  taken  by  NPS  Rascal  UAS  while 
controlled  by  medic  via  the  voice  commands. 


The  Special  Forces  battlefield  medic  managed  to  direct  the  NPS  UAS  to  the  casualty  site,  to 
inject  medication,  to  control  the  surveillance  camera  ,  and  to  record  observations  into  the 
experiment  Observer  Notepad,  all  by  successfully  “talking”  to  the  UAS  and  sensors  via  the 
CENETIX  Voice  Portal  using  a  standard  Blackberry  handheld  with  a  head  set. 

The  results  demonstrated  sufficient  accuracy  for  controlling  the  tactical  sensor-robotic  assets  via 
the  VoT  interface: 

•  4  of  4  drug  deliveries  of  injected  liquid  drug  into  a  vial  on  the  casualty  mannequin  were 
successfully  completed, 

•  5  out  of  5  voice  comments  were  successfully  recorded  in  Observer  Notepad  over  the 
CENETIX  Voice  Portal, 

•  1  out  of  1  voice-controlled  camera  pointing  command  was  successfully  sent  over  the 
Voice  Portal, 

•  10  out  of  10  voice  commands  for  the  voice-controlled  camera  were  successfully  sent  to 
adjust  camera  orientation, 

•  4  out  of  5  tasking  commands  were  successfully  sent  and  executed  to  call  the  NPS  UAS  to 
the  casualty  location  for  taking  high  resolution  imagery.  One  of  voice  commands  did  not 
go  through,  possibly  caused  by  poor  GSM  coverage  in  the  area  of  operation,  and  was 
substituted  for  by  sending  a  command  over  the  SA  Agent  Web  interface. 


3.  Field  Experiment:  Applying  VoT  to  Precise  Parafoil  Descending  Control 

Another  example  of  successful  voice  interface  implementation  in  robotic  system  control  is  a 
precise  parafoil  landing.  The  experiment  was  addressing  small  payload  precise  landing  (with  10 
meters  accuracy)  to  deliver  equipment,  medicine,  and  such  to  the  area  of  operation  by  manned  or 
unmanned  aircraft.  Precise  landing  may  be  accomplished  by  periodic  updates  of  the  target’s 
local  weather  condition  information  by  sending  it  to  the  control  algorithm  while  the  parafoil  is 
descending.  The  network  and  Voice  Portal  allowed  monitoring  and  descending  parameters 
adjustments  from  a  remote  location  .  The  network  utilized  Voice  Portal  Client  to  assign  target 
coordinates  over  regular  GSM  handheld  (Blackberry  phone)  as  shown  in  Figure  9.  The  GSM- 
enabled  parafoil  payload  and  weather  station  with  voice  control  utilization  are  presented  in 
Figure  10.  For  this  field  experiment  two  payloads  with  parafoils  were  attached  to  an  manned 
aircraft’s  canopy  and  delivered  to  the  area  of  operation.  The  command  to  drop  the  parafoil  was 
send  by  voice  command.  When  the  parafoil  got  unfolded  in  the  air,  another  voice  command  was 
sent  to  provide  the  exact  landing  GPS  position.  Voice  commands  were  sent  from  the  regular 
GSM  cellular  phone  (Blackberry  phone)  by  placing  the  call  and  following  the  Voice  Portal 
instructions  (Figure  2). 
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Figure  9.  Voice  control  enabled  parafoil  network  diagram. 


Figure  10.  GSM-enabled  parafoil  payload  and  weather  station  with  voice  control  utilization 


Conclusion 


The  described  VoT  solutions  represent  good  examples  of  an  alternative  approach  to  sharing 
situational  awareness  information  between  man  and  machine  in  the  tactical  network-centric 
environment  augmented  by  unmanned  robotic  systems.  It  takes  us  significantly  closer  to  the  long 
time  anticipated/desired  seamless  natural  language  with  robots  in  the  battlefield.  The  VoT 
approach  also  provides  for  a  unique  capability  for  “humanizing”  the  unmanned  systems  data 
exchange  by  enabling  mapping  data  transfer  commands  into  the  voice  commands.  For  example, 
the  tactical  operations  commander  could  literally  hear  the  unmanned  systems  “talking”  (voice 
report)  back  to  him,  providing  current  status  of  task  execution.  This  in  turn  enables  the 
commander  to  start  “sensing”  unmanned  systems  networking,  thus  improving  the  commanders 
cognition  and  situational  understanding.  We  would  like  to  emphasize  that  GSM  network 
utilization  also  naturally  provides  another  attribute,  global  reach. 
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Overview 


•  Motivation  and  Background 

•  VoT  Concept  and  Voice  Portal  architecture 

•  GSM  service  utilization  in  Tactical  Network 

•  Voice  C2  in  Tactical  Operations 

•  Extending  GSM  coverage 

•  Utilization  examples  and  live  demo 


Motivation 


Even  the  most  experienced  operators  have  no  chance  to  concentrate  on 
opening  and  viewing  a  computer  screen  with  the  map-based  situational 
awareness  interface  while  in  the  operation  involving: 

>  Small  vessel  interdiction  at  high  speeds  of  30-50  knots 

>  The  remote  control  of  Unmanned  Aerial  Vehicles  (UAVs), 
while  on  the  fast  move  through  the  rugged  terrain 

>  Casualty  assistance,  while  still  in  the  hostile  area 

>  Sniper  operations 

Unlike  a  visually-rich  common  operational  picture  tool,  voice  communication 
interferes  substantially  less  with  an  operator’s  ability  to  carry  out  a 
concurrent  task. 

This  is  one  of  the  few  feasible  solutions  for  tasking  unmanned  systems 
while  keeping  hands  and  eyes  free  for  more  immediate  tasks. 


USSOCOM  -  NPS 

Field  Experimentation  Cooperative 


•  Networks  (wireless,  SATCOM,  IPv6,  etc.) 

•  Unmanned/autonomous  vehicles  -  collaborative  behavior,  management, 
payloads 

•  Networked  sensors 

•  Situational  awareness  (SA)/Common  operational  picture 

•  Biometrics  and  forensics  with  link  analysis 

•  Collaboration  technologies  with  reach-back 

•  Human  systems  performance  and  integration 

•  I  ED  detectors  and  jammers 

•  Decision  making 

•  Modeling  and  simulation  for  mission  planning  and  optimized  asset  utilization 

Example  Scenarios 

•  Surveillance  and  reconnaissance  •  Sniper 

•  Tagging,  tracking,  and  locating  missions  •  Battlefield  medical 

•  Detection  and  interdiction  of  small  craft  with  nuc/rad  •  Downed  pilot 

•  Riverine  operations  •  Security  checkpoint 

•  Simultaneous  multiple-site  collaboration  and  SA  with  actionable  intelligence 


jUL  USSOCOM  -  NPS 

Field  Experimentation  Cooperative 


Monterey  Bay,  Pacific  Ocean 


NSWC  Dahlgren,  VA 


NSW  Coronado,  CA 


NAWCWD,  China  Lake 


J  U.C.  Santa  \' 


!  Barbara 


802.16 


-100  mi 


Santa  Barbara 
County  Police  and 
Fire 


Ft.  Hunter 
Liggett,  USAR 


Camp  Roberts 


ANG 

NPS/CIRPAS  McMillan 
Field  UAS  Flight  Facility 


Pacific 

Ocean 


NSWC  Group  4 

LMCO  Center  for  Innovation 

VA 


Sea  Fox  USV 


Mt.  Diablo 


San  Francisco  Bay 


U.S.C.G.  Yerba 
Buena  Island,  CA 


802.16 


Self-aligning  802.16 


NPS  CENETIX 


LBNL 


USSOCOM  -  NPS 

Field  Experimentation  Cooperative _ 

Plug-and-Play  MIO  Testbed 


Riverine  - 

Sacramento 

Delta 


Port  of  NY/N  J 


Bay  /  Port 


Austria,  Germany,  Sweden,  Denmark,  Greece,  UK,  Turkey 


^jNPsy. 


CENTER 

OR  H&TWQRK 


CENETIX 


Self-Organizing  with  Tactical  Network 


Network-on -Target  Demo 
(designed  by  E.  Bourakov) 


nnovatjqn  AND 
XPEQJME  N  TAT  iOn 


SAOFDM  MIO.swf 


Enabling  Network-Centric  UAV,  Biometrics  and 
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Research  Partnerships  with  International  Participants 

Maritime  Interdiction  Operations 


Objective 

Evaluate  the  use  of  networks,  advanced  sensors,  and 
collaborative  technology  for  rapid  Maritime  Interdiction 
Operations  (MIO),  Port  Security,  and  Riverine 
Operations  - 

The  ability  for  a  Boarding  Party  to  rapidly  set-up  ship- 
to-ship  communications  that  permit  searching  for 
radiation  and  explosive  sources  and  conducting 
biometric  ID  while  maintaining  network  connectivity 
with  C2  organizations,  and  collaborating  with  remotely 
located  sensor  experts,  coalition  partners,  and  first 
responders. 


Example  Technologies 

•  Innovative  Wireless  Networks  and  Sensors 

•  SATCOM  on-the-Move  and  Orbital  Ad-Hoc  Networking 

•  Drive-by  Radiation  Detection  of  Large  Ships  and  Small  Boats 

•  Projectile-Based  Wireless  Links 

•  Networked  USVs  and  UGVs 

•  Collaboration  and  Decision  Making 

•  Situational  Awareness 

•  IPv6  Sensors-on-the-Move 

•  Environmental  Effects  on  Target  Detection,  Comms,  and  Plume  Dispersi 

•  Forward  Deployed  Biometrics  with  Reach-Back 

•  Tagging  and  World-Wide  Monitoring 


Adding  Unmanned  Systems  to  MIO  Network:  Drive-by  Search  by 


USV,  UAV  Relay  to  the  Fast  Boat,  UGV  in  the  Tunnel 


USV  provided  radiation  detection  in  small-boat  drive-by  with  real¬ 
time  expert  reachback;  network-controlled  USV  &  UGV 


Lockheed  -  133 


MIO  08-4  Days  3  &4,  Ft.  Eustis,  VA:  Small  craft 
stand-off  detection  and  interdiction  at  hiqh  speeds 
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VoT  Concept  and  Voice  Portal  architecture 
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Extending  GSM  coverage 


ubiquitous  control  link 
two  way  communication 
positioning  information 


Tactical  Cellular  Network: 

•  fast  GSM  infrastructure  deployment 
a  •  commercial  GSM  network  extension 
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Utilization  examples 
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Situational  Awareness  user  interface  for  mission  tasking. 


The  UAV  waypoint  for  taking  high  resolution  image  is  assigned  by  voice  command. 


Utilization  examples 


Two  armed  “Snowflake”  units 
ready  for  deployment  and 
control  over  the  Voice  Portal 
In  Camp  Roberts,  CA 


Fully  deployed  “Snowflake” 
and  its  payload. 


Utilization  examples 


“Snowflake”  deployment  network  infrastructure  with  voice  control  utilization 


Summary 

Alternative  approach  to  sharing  situational  awareness  information  between 
man  and  machine  in  the  tactical  network-centric  environment 


Long  time  anticipated  seamless  natural  language  to  communicate  with 
unmanned  systems  in  the  battlefield. 

“Humanizing  ”  the  unmanned  systems  data  exchange  by  enabling 
mapping  data  transfer  commands  into  the  voice  commands. 

Tactical  operation  commander  could  literally  hear  unmanned  systems 
“talking”  back  to  commander  providing  voice  report  of  current  task  execution 
status. 


VoT  approach  enables  “sensing”  unmanned  systems  networking  and 
improve  operator's  cognition  and  situational  understanding. 

GSM  network  utilization  naturally  brings  very  much  desirable  in  many 
operations  capability  -  a  global  reach. 


Questions 


Eugene  Bourakov 
ebourako@nps.edu 
(831  )-91 5-5688 

Dr.  Alex  Bordetsky 
abordets@nps.edu 
(831  )-91 5-2408 


